Load balancing in set top cable box environment

ABSTRACT

A scalable messaging system for data transmission between the network devices, such as set top boxes, and a central system server, such as a server which maintains a database of event logs for the network. Individual routers at the data center broadcast an announcement packet indicating that they are available to accept messages from the network devices. The announcement message contains at least an identification of the router and the manner in which messages may be sent to it, e.g., one or more connection socket numbers and/or network addresses. The frequency at which availability messages are sent by the routers is preferably dependent upon the relatively loading of the individual router. Thus, the more heavily loaded a particular router becomes, the less often it will broadcast an availability message; the more lightly loaded it becomes, the more often such messages are broadcast. The network devices then transmit messages to the data center only in response to having received such a router availability announcement. The information in a router availability message can be used in various ways to construct a payload message back to the data center, such as by using ports numbers, persistent identification numbers, or Media Access Control (MAC) layer addresses, depending upon the topology of the data network. This protocol thus permits control over the generation of messages, such as connection request messages, which might otherwise flood a network with large numbers of end node devices.

RELATED APPLICATION(S)

[0001] This application claims the benefit of U.S. ProvisionalApplication Ser. No. 60/253,442, filed on Nov. 28, 2000, the entireteachings of which are incorporated herein by reference.

BACKGROUND OF THE INVENTION

[0002] At the present time, most data network devices located inresidences include some type of personal computer. Typically, thesepersonal computers are used to connect to Internet Service Providersover dial-up connections to execute application programs such as emailclients and Web browsers that utilize the global Internet to access textand graphic content. Increasingly, the demand is for multimedia content,including audio and video, to be delivered over such networks. However,the backbone architectures of purely data networks, especially thosedesigned for use with the telephone network, were not originallydesigned to handle such high data rates.

[0003] The trend is towards a more ubiquitous model where the networkdevices in the home will be embedded systems designed for a particularfunction or purpose. This has already occurred to some degree. Today,for example, cable television (CATV) network set-top boxes typicallyhave limited data communication capabilities. The main function of thedata devices is to handle channel access between residential users and ahead end or server on the cable TV network.

[0004] However, it is estimated that the worldwide market for Internetappliances such as digital set-top boxes and Web-connected terminalswill reach $17.8 billion in 2004, and millions of such digital set-topboxes have already been deployed. Increasingly, advertisers and contentproviders view the cable set-top as the first platform of choice forwidespread delivery of a suite of intelligent content management anddistribution services.

[0005] In the future, the functionality offered by these set-top boxesor other embedded platforms, such as a game system, will be expanded.For example, they may offer Internet browsing capabilities ande-commerce serving capabilities. Moreover, it is anticipated thatcommon-household appliances will also have network functionality, inwhich they will be attached to the network to automate various tasks.

SUMMARY OF THE INVENTION

[0006] Because of their extremely large number of network devices insuch networks, efficient distribution and delivery of managementservices, promotions and digital content remains a challenge. The datanetworks must evolve with deployment of these embedded systems. Wherethe personal computer can be updated with new network drivers as thenetwork evolves, embedded client systems remain relatively static. Suchnetworks may have hundreds of thousands, if not millions, of networkdevices to manage. It is evident that standard data Open SystemsInerconnection (OSI) layered network protocols, which were optimized forpeer-to-peer communication, are not an entirely acceptable arrangement.

[0007] Consider that the digital set top box provides certaininteresting functionalities, such as the ability to collect data, suchas a log of the channels watched over time, and other events. The settop box can be designed and program to them report this information to acentral location. At the central location, this data can be aggregatedfor many hundreds of thousands of users. This information, when coupledwith other information such as demographics, can then be used byadvertisers and service providers to blanket defined market segmentswith promotions, advertisements, and content. The digital delivery ofpromotions can then allow for impulse responses yielding immediateincreases in revenues.

[0008] However, such a network may have hundreds of thousands, if notmillions of set top boxes, attempting to deliver event log information.If such a data network were built using standard data protocols such asTCP/IP, the sheer number of connection request messages alone couldcause the performance of a central data server to degrade to the pointwhere it is unable to carry out reliable message delivery.

[0009] The present invention implements a scalable messaging system fordata transmission between the network devices, such as set top boxes,and a central system server, such as a server which maintains a databaseof event logs for the network.

[0010] Specifically, the individual routers at the data center broadcastan announcement packet indicating that they are available to acceptmessages from the network devices. The announcement message contains atleast an identification of the router and the manner in which messagesmay be sent to it, e.g., one or more connection socket numbers and/ornetwork addresses.

[0011] The frequency at which these availability messages are sent bythe routers is preferably dependent upon the relative loading of theindividual router. Thus, the more heavily loaded a particular routerbecomes, the less often it will broadcast an availability message; themore lightly loaded it becomes, the more often such messages arebroadcast.

[0012] The network devices then transmit messages to the data centeronly in response to having received such a router availabilityannouncement. The information in a router availability message can beused in various ways to construct a payload message back to the datacenter, such as by using ports numbers, persistent identificationnumbers, or Media Access Control (MAC) layer addresses.

[0013] This protocol thus permits control over the generation ofmessages, such as connection request messages, which originate at thenetwork devices. It avoids a situation whereby such messages mightotherwise tend to flood a network that consists of an extremely largenumber of end nodes that need to communicate to a central location.

[0014] The implementation of a device management protocol in this mannerassists network operators to cost effectively support the advancedfeatures of the set-top box, such as to provide targeted promotion anddigital content distribution services. This enables network operators togenerate new revenues and provide a richer interactive environment forconsumers.

BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The foregoing and other objects, features and advantages of theinvention will be apparent from the following more particulardescription of preferred embodiments of the invention, as illustrated inthe accompanying drawings in which like reference characters refer tothe same parts throughout the different views. The drawings are notnecessarily to scale, emphasis instead being placed upon illustratingthe principles of the invention.

[0016]FIG. 1 is a block diagram of a network in which a messagingprotocol according to the invention may be used to control thetransmission of messages from an extremely large number of transmittingnetwork devices to a central receiver location.

[0017]FIG. 2 is a high level process flow diagram of a particularapplication which makes use of the protocol to deliver targetedpromotions and content.

[0018]FIGS. 3A and 3B are a process flow diagram illustrating how arouter processes different types of messages.

[0019]FIG. 4 illustrates a Global Unique Identifier assigned to thenetwork devices so that they may be addressed by an application.

[0020]FIG. 5 is a high level router state diagram.

[0021]FIG. 6 is a high level state diagram for how a router handlesnetwork device connections.

[0022]FIG. 7 is a high level message state diagram.

[0023]FIG. 8 is an exemplary router announcement service message.

[0024]FIGS. 9A, 9B, and 9C are different types of device connectionmessages returned in response to the router announcement servicemessage.

DETAILED DESCRIPTION OF THE INVENTION

[0025] 1. A Targeted Promotion Delivery System

[0026] Turning attention now to the drawings, FIG. 1 illustrates amultimedia content delivery system which uses a message routing protocolaccording to one embodiment of the present invention. The systemincludes a large number of set top boxes or network devices 10 connectedto respective video displays 20, such as televisions. Promotions 30typically include promotional content that may be presented in variousmultimedia formats including standard audio visual clips, but alsocomputer graphics, icons, or Hypertext Markup Language (HTML) content.Promotions are used to advertise goods and services, promote events, orpresent other commercial or non-commercial information. One or morepromotions 30 may be simultaneously active within the video displays 20and may be displayed in different ways. For example, promotions 30 canbe presented on electronic program guides, channel information bars, orby overlaying video broadcast programming. Some active promotions thatmay be displayed on digital set top boxes allow user interaction such aslinking to e-commerce web-sites via hyperlink connections or directcommunication with the server subsystem of the promotion delivery systemto obtain additional software, such as device drivers, video games, orother application software.

[0027] As shown in FIG. 1, the network devices also include a promotionserver subsystem 200 located at a data center, and a promotion agentsubsystem 300 embedded within each of the network devices. The promotionserver subsystem 200 and the promotion agent subsystems 300 communicatewith each other through a combination of application-level messaging andserialized bulk data transmissions.

[0028] The promotion server subsystem 200 periodically collects viewerusage data from the promotion agent subsystem 300 of each of themultimedia content viewing devices to generate viewership profiles. Intelevision networks, the data collected by the promotion serversubsystem 200 may include tuner data (i.e., a history of channelswatched) and responses to past promotions. This history is kept on arelatively fine time scale, such as five seconds. In this way, it can bedetermined how long a particular promotion was deployed, or even whichportions of a promotion or video program were viewed.

[0029] In more detail regarding promotion delivery, the promotion serversubsystem 200 includes a database server 210, a promotion manager server220, a bulk data server 230, a promotion manager client 240, alife-cycle manager server 240, and a bank of routers 250-1, 250-2, . . ., 250-n, and a queue manager 260. These components are typically locatedat a central location in the multimedia network at a data center, at ahead end, or divided between the two depending on the density andpopulation of devices. It should be understood that these components mayshare physical platforms or de distributed across multiple machineslocated at different places in the network. For scalability reasons, apromotion packaging process in the promotion manager server 220 may beseparated from a function which is responsible for delivering promotionpackages to the network devices 10. The delivery function may beinstantiated on multiple machines, for example, to provide betterscalability, such as having one machine per head end in the network. Thelife cycle manager 240 may also be instantiated separately for eachrouter 250.

[0030] The routers 250 communicate with the network devices 10 through adata network 75 which may itself include a further hierarchy of routersand bulk servers (not shown in FIG. 1). Ultimately, each of the networkdevices is connected to the network 75 through a head end location 50.In a typical cable television network, there may be many thousands ofnetwork devices connected to a particular head end, and there may bemany thousands of head ends 50.

[0031] The queue manager 260 is provided for facilitating the transferof messages between the message routers 250 and the other systemcomponents. The queue manager 2600 is an application-level process thatcommunicates with the message routers 250 on behalf of other processes,such as the promotion manager 220, or the promotion agent in the networkdevice 300, in order to send and receive messages among them. In oneembodiment, the queue manager 260 is implemented as a C++ object. Thequeue manager 260 also manages incoming and outgoing messages queues onbehalf of the processes in the system process running at the data center200.

[0032] The queue manager 260 handles two types of queues, persistentqueues and volatile queues. Messages, whose message type indicatespersistent storage, are stored such that the message will not be lostduring power outages and lost network connections. A persistent queue isstored in persistent flash memory or in a location on the hard disk ofthe network device. Other messages, not intended for persistent storage,are stored to volatile queues and might be lost during power outage andlost network connections.

[0033] To determine how to deliver targeted promotions to the networkdevices, the life-cycle manager server 240 of the promotion serversubsystem 200 first generates viewership profiles for each of themultimedia content viewing devices from the collected data using avariety of statistical models. The viewership profiles are then used toassociate groups of network devices with a given target promotion.

[0034] Promotion groups are collections of multimedia content viewingdevices whose individual viewership profiles match membership criteriondescribing a particular demographic or viewership history. For example,a promotion group may be demographically based, i.e., “married women intheir 30's with more than one school age child and a household income ofat least $ 100,0000,” or based on viewership history, i.e., “tends towatch the Golf Channel on Sunday afternoon.” Therefore, the promotiondelivery system is adaptable to changes in viewer usage or viewershippatterns by making adjustments to promotion groups. Promotion groups aredescribed in more detail in U.S. Provisional Patent Application Ser. No.60/253,488 filed Nov. 28, 2000, entitled “Using Viewership Profiles forTargeted Promotion Deployment” which is hereby incorporated by referencein its entirety.

[0035] Promotions are then scheduled for delivery to specific promotiongroups. A promotion is scheduled for delivery to a promotion group by anadvertiser or service provider entering a scheduling request for apromotion such as via a promotion manager interface client 225. Thepromotion manager server 220 packages the promotion for delivery andstores it in the database 210. Later, the package information is readfrom the database 210 and used to create customized transmissionschedules that specify when and how each of the network devices 10 is toreceive it. A preferred technique for packaging promotions into messagesto be sent to the network devices is described in U.S. Provision PatentApplication Ser. No. 60/253,489 filed Nov. 28, 2000, entitled “PromotionPackaging for Transmission Groups” which is hereby incorporated byreference in its entirety.

[0036] The promotion agent subsystem 300 embedded in each of the networkdevices 10 includes a promotion agent 310 and a bulk data agent 320.Upon receipt of the transmission schedule messages, the promotion agent310 processes each schedule entry and waits for the bulk data agent 320to deliver each promotion identified in the transmission schedule. Thebulk data agent 320 then handles the reception of the promotions fromthe scheduled data transmission as specified in the promotion downloadrequests. For example, in one embodiment, the bulk data agent 320 tunesinto a multicast data transmission stream at a specified time andchannel or network address specified in the transmission schedule.

[0037] The promotion manager server 220 extracts the promotion packagefrom the database 210 and converts it into a transmission request thatis sent to the bulk data server 230. The bulk data server 230 fetchesthe promotions from the database 210 that are identified in thetransmission request message, and transmits them via multicast orbroadcast transmission depending on transmission control data specifiedin the transmission request.

[0038] Once the promotions have been successfully delivered, thepromotions are activated at the network viewing devices as specified inpromotion control data of the transmission schedules. Promotionactivation may be event, time, or channel driven.

[0039]FIG. 2 illustrates a generalized process diagram 400 for creatinga viewership profile of a viewer who has tuned to a program channel on anetwork device 10. In a first step 402, the promotion agent 310 of thepromotion agent subsystem 300 embedded in the network device 10 createsan event log of the viewer's activities. The event log records thechannel to which the set top box is tuned to, the time the channel wastuned in, and the time the it left the channel. In the describedembodiment, the event is recorded only if the period between the timethe viewer tuned in the channel and the time the viewer tuned away fromthe channel is greater than about five seconds. By logging events thathave only been watched for a period greater than five seconds, thepromotion agent is able to distinguish shows that are actually watchedfrom channel “surfing” by the viewer.

[0040] After the promotion agent 310 has logged viewer activities for aperiod time, such as twenty four hours, the logged activities aretransmitted through messages, in a state 404, to the life cycle managerserver 250. The messages are transmitted through a messaging protocolfor unicast transmission, such as TCP/IP.

[0041] It is important to note here, however, that the uploading ofthese messages does not occur simply at the whimsy of the promotionagent 310 in the network device. Rather, a specific protocol is used bythe system whereby the routers 250 advertise their ability to acceptmessages from the network devices 10, and the end nodes only attempt tocommunicate with the data center in response to receiving such messages.

[0042] In a state 406, the life cycle manager receives the event logfrom the promotion agent 310. Also, in the state 406 a program schedule260 is periodically transmitted to the life cycle manager server 250.Such program schedule data for broadcast network is typically availablefrom commercial services.

[0043] After receiving the logged viewership activities and the programschedule 260, the life cycle manager server 250 correlates the data inthe state 406. Here, the life cycle manager determines the viewerbehavior associated with the network devices. The life cycle manager mayfor example, determine what programs were watched and the percentage oftime each program was watched during its scheduled time slot.

[0044] The viewer behavior data generated by the life cycle managerserver is matched with group profiles 270 provided by third parties,such as advertisers, to the life cycle manager server 250. These groupprofiles 270 may include age, gender, residence and other demographicdata.

[0045] Subsequently, in a state 408, the matched viewership behaviordata and group profiles 270 are used to select and then download atargeted promotion to the determined class of the viewer. In a state410, this promotion is delivered to the network devices 10.

[0046] 2.0 An Overview of Router Functionality

[0047] Before continuing with a discussion of the protocol used toeffect the delivery of event log information from the network devices 10to the life cycle manager in step 404, it is illustrative to considerthe routers 250 in more detail.

[0048] As mentioned previously, messages are delivered to and from thedata center and the network devices through the routers 250. Messagescome in two flavors: application and control. The application messagesdeliver data content; control messages are used to co-ordinate delivery.Application messages can have one of two delivery methods: Datagram andStandard. Standard messages guarantee persistence via a receipt controlmessage. A message receiver sends a receipt to the sender for one ofthese messages as soon as the receiver has guaranteed that the messagepersists somewhere upstream of the sending device. Receipting orpersistence functions are not performed for datagrams.

[0049] Each router 250 generally implements a protocol as follows:

[0050] It checks to see if the destination for a message is online.

[0051] If the device is online, the router forwards the message to thatdevice and waits for the recipient to receipt the message.

[0052] If the recipient receipts the message within the time allotted,then the recipient has guaranteed that the message persists upstream ofthe router. The router then sends a receipt to the message's sender ifrequired.

[0053] If the recipient does not receipt the message, the routerpersists the message in the database. The router will periodicallyattempt to deliver the message until the message expires or until therecipient sends a receipt.

[0054] If the network device is offline, the router persists the messagein the database and attempts to bring the device online. The router willsend the message to the destination device when it comes online and willattempt to deliver the message periodically until the message expires oruntil the recipient sends a receipt.

[0055]FIGS. 3A and 3B illustrate a generalized process flow diagram forthe handling of messages by the routers 250.

[0056] In a first step 1101, a router 250 receives a message either froma device 10, from the database 210 or from another router 250 using arouter-router protocol.

[0057] In step 1102, if a delivery method message parameter indicatesthat the message is to be sent as a datagram, i.e, indicates that themessage should not be persisted if currently undeliverable.

[0058] In step 1103, the router sends a receipt control message to thesending entity (if the message is from a network device). The receiptobject is sent back to the originating sender if using the router-routerprotocol.

[0059] In step 1104, the destination identifier in the message isexamined. This will typically contain a network device identifier.However, one special case exists where the destination deviceID=HG_HYPERGATE is used. This indicates a message to be sent to thecomponents of router's internal machine itself, such as its messagequeue. The router chooses a recipient from its list of registeredservices and constructs the queue name and destination ID needed toroute the message to that service.

[0060] In step 1105, a filter function is performed that removesimproperly addressed router messages. The only acceptable value is“Router”.

[0061] In step 1106, the message is handled as if sent to the router'squeue via the internal router queue manager.

[0062] Step 1107 is reached if the message had an improper queue name,at which point the message is discarded.

[0063] Step 1108 is reached if the message did not have the internalrouter identifier. In this step, it is determined whether the router IDportion of the device specified in the destination ID matches therouter's router ID.

[0064] Step 1109 checks to see if the device indicated by the device IDportion has a TCP connection to this router.

[0065] In step 1110, the message is sent to the device if connected.

[0066] Step 1111 checks to see if the router should discard a datagramsent to a device presumed to be offline.

[0067] Step 1112 persists standard messages to devices presumed to beoffline. If the specified device is online with another router, thatrouter will get the message via the database.

[0068] Step 1113 discards datagrams to offline devices.

[0069] Step 1201 is reached if the message is not intended for therouter itself. First, the router checks to see if it is connected to therouter indicated in the router ID portion of the device indicated in themessage's destination ID.

[0070] In step 1202, if the device is connected, the router sendsmessage to one of the other routers 250 presumed to be the path to thespecified device (a device could switch routers during the periodbetween message creation and message delivery).

[0071] At step 1203, if not connected, the router asks the database forthe port and P address on which to make a router-router connection tothe destination device's router.

[0072] In step 1204, check the database's query for a path to thedestination device's router.

[0073] In step 1205, if no such path exists, then persist message todatabase. If the device is online, its router will get the message fromthe database and deliver it.

[0074] In step 1206, a TCP connection is made to the destinationdevice's router.

[0075] Step 1207 checks for a connection success or failure (due totimeout, network error, etc.).

[0076] In step 1208, the router persist the message to the database ifthe connection could not be established.

[0077] In step 1209, if the connection succeeded, the message is sent tothe destination device's router via the router-router protocol.

[0078] 2.1 Router to Router Extensions

[0079] Router to router protocol extensions are implemented to permitthe routers 250 to communicate with each other. This protocol followsthe same basic principle as router/device communication. A receipt by arouter indicates that the message has persisted somewhere upstream. Ingeneral, all routers try to forward messages outside of the database,but some database method of persistence has to be available in case theend device is offline.

[0080] Router to router communication is different from device to routercommunication. In general, routers should always be online. Also,routers are a trusted entity within the system and have a lessrestricted network path to other routers. Router communication istailored to these considerations. Routers are able to establish aprivileged connection with other routers in order to forward messages.

[0081] This router-to-router protocol permits the routers to cooperatein order to coordinate the following tasks:

[0082] Device connection—the system provides centralization of devicestate within the database which maintains information as to which routeron a head end connects to a particular network device 10. The routers250 recognize the database information as correct and synchronized.

[0083] Message exchange—the system also provides a mechanism other thanthe database for forwarding messages from a service attached to onerouter to a device attached to a different router and vice-versa.

[0084] Message persistence—the system also provides a mechanism forpersisting messages to offline devices.

[0085] Service location—the system has a mechanism that allows devicesto send messages to a service without knowing, a priori, which machinehosts the service.

[0086] Router performance—the system is able to judge router load andmaintains some indication as to whether the router is functioningproperly.

[0087] 2.2 Device Representation

[0088] Routers 250 in a multiple router system need to be able toassociate a particular network device 10 with the routers that canconnect or are connected to the device. This information is localized ina Global Unique Identifier (GUID) assigned to each network device 10.The use of a GUID permits the application level processes to identifydestination devices without the need to maintain information as to thespecific types of transport in use, or a device's network address. Thedevice GUID hold two pieces of routing information, the network ID andthe router ID. The network ID represents the set of routers that canconnect to a given device. The router ID represents the particularrouter in a network that is currently connected to a device. Each routerhas a unique combination of network and router ID information.

[0089] Devices have a device ID that uniquely identifies them. Eachdevice also has a network ID that identifies the sub-network head towhich the device is connected. The network ID is not necessarilypermanent, since head end configurations may change, but the network IDshould persist with the same value for a long period of time. Thenetwork ID could also be a head end ID, but using network IDsaccommodates a situation where multiple head ends are located in thesame sub-network. Each connected device also has a router ID thatidentifies the router that is attached to it. Together, the device ID,network ID and router ID make up the device GUID, as shown in FIG. 4.

[0090] A service sending an unsolicited message to a device must get thedevice ID from some location; typically the database. A function isprovided in the database that generates a device's device GUID given thedevice's device ID. Typically, stored procedures will use the device IDto join tables to the device table, but will write out the device GUIDwhen assembling the final output. A device might not be online when thedevice GUID function is called. The device GUID function will specify arouter ID if none is currently specified and will mark the device aseligible to go online if it is not currently online. The systemanticipates that a request for a device GUID indicates that a messagewill soon be sent to it and tries to prepare the device appropriately.

[0091] The device GUID function will contain load balancing logic. Adevice should be associated with the last router to service it, forconsistency. A device should be associated with the router that has theleast load. The device GULID will weigh these two considerations,reassigning a device if its former router is offline or is experiencinga load that degrades its performance significantly in comparison toother routers on the head end.

[0092] 3.0 Router Database Tables and Procedures Detail

[0093] This section documents database tables and stored procedures usedby the routers 250.

[0094] 3.1 Router Database Tables

[0095] Table 1 is a database table T_NETWORK that describes a network.Examples of networks are the data center, the network that the MultipleSystem Operator (MSO) exposes for control of devices, and the Internetat large. TABLE 1 T_NETWORK Column Type Meaning or value ID NumberPrimary key for the network NAME String User-readable, descriptive namefor network SECURE1 Number 0-network is open to the Internet on Navic'sports. 1-network is open to MSO's devices 2-network traffic limited tothe data center on Navic's ports. KBITS_PER_SECOND Number # of kilobitsthat can be transmitted over the network per second (currently not used)MULTICAST_TTL NUMBER # of router hops for multicast transmissions overnetwork BROADCAST_TTL NUMBER # of router hops for broadcasttransmissions over network NET_MASK NUMBER IP mask for network's addressspace MC_AVAILABILITY_(—) NUMBER IP address to use to multicast ADDRESSrouter availability MC_AVAILABILITY_(—) NUMBER Port to use to multicastrouter PORT availability LISTENING_ROUTER_ID NUMBER Router that'scurrently (fk of T_ROUTER.ID) listening for device connect requestsMC_AVAILABILITY_(—) NUMBER # of (fractional) days between FREQUENCYmulticast availability trans- missions.

[0096] Table 2, T_ROUTER, represents a router servicing devices or aservice that makes connections to the router via the router-routerconnection. TABLE 2 T_ROUTER Column Type Meaning or value ID NUMBERPrimary key WATCHDOG_TIME Date Last time router registered usingSP_HGS_WATCHDOG LOAD_METRIC Number Metric indicating router's load-higher indicates router is stressed. STATE Number Routerstate-STATUS_OFFLINE - router is offline. STATUS_ONLINE-router isprocessing messages and connections. STATUS_ONLINE_DISCON- NECT - routerhas not met its watchdog time and has been marked to be taken offlineDNS_NAME String Router's host name. DEVICE_ID Number A device ID thatcan be used to talk (fk of T_DEVICE) to the queue manager on therouter's machine. SERVICE_TYPE Number 0-a router other-service type of aservice connecting via the router-router connection

[0097] Note in particular from the above that each router periodicallydetermines a relative load metric and stores this information in theLOAD_METRIC entry in the table. In the preferred embodiment, a lowernumber indicates a better performing router. As will be understoodshortly, the LOAD_METRIC entry is used by the router to determine howoften to send an availability message to the network devices 10.

[0098] T_ROUTER_NIC, Table 3, represents a network card in a router.Typically, a router will have a network card with an IP address in thedata center's firewalled network and one or more cards with IP addressesin an MSO's network. TABLE 3 T_ROUTER_NIC Column Type Meaning or valueNIC_INDEX Number Zero-based index of the network interface card in therouter's IP address table. ROUTER_ID Number Router ID of router inquestion (fk of T_ROUTER.ID) NETWORK_ID Number The ID of the network towhich (fk of T_NETWORK.ID) the card is attached. IP_ADDRESS Number Thebind address to be used by IP to talk to the card DEVICE_PORT Number Theport # to bind to listen for connect requests from devices ROUTER_PORTNumber The port # to bind to listen for TCP connect requests from otherrouters

[0099] Table 4, T_ROUTER_NIC_IN, is populated on initialization. Ittells the stored procedure, PKG_HGS_ROUTER.SP_HGS_INIT, what networkcards exist on the router. TABLE 4 T_ROUTER_NIC_IN Column Type Meaningor value TRANSACTION_GUID RAW(16) All rows specific for a particularinvocation of SP_HGS_INIT are identified by the router by this GUID.NIC_INDEX Number The zero-based network card index of the cardassociated with this row. ROUTER_ID Number The ID of the router hostingthe network card IP_ADDRESS Number The IP address to use when binding tothis card.

[0100] Table 5, T_ROUTER_TUNING, contains one row that maintains therouter tuning parameters used in the database. These parameters are usedin the stored procedures. TABLE 5 T_ROUTER_TUNING Column Type Meaning orvalue LOAD_METRIC_THRESHOLD Number SP_HGS_ONE_(—) CONNECT_REQUEST willreassign devices connected to routers whose load metrics are above thisthreshold. WATCHDOG_TIMEOUT Number Maximum allowable # of (fractional)days between calls to SP_HGS_(—) WATCHDOG before a router is takenoffline (a router must call SP_HGS_WATCH- DOG at least this often or itwill be taken off- line) MSG_SEND_TIMEOUT Number Number of (fractional)days before a message is resent to a device that's online, but hasn'tresponded to a previous message send. MSG_CONNECT_TIMEOUT NumberMeasured in (fractional) days. The router will bring a device online ifit receives a message for the device and if it it was able to bring thedevice online last time. If the router failed to bring the device onlineon the previous attempt, it will not attempt again unless the attemptwas MSG_CONNECT_(—) TIMEOUT days ago. CONNECT_TIMEOUT Number Measured in(fractional ˜15 minutes) days. The router reconnects if it receives aconnect request for an online device if the connection was establishedmore than CONNECT_(—) TIMEOUT days ago. CONNECT_REQUEST_TIMEOUT NumberMeasured in (fractional ˜30 sec.) days. The router will ignore a secondconnect request if it was recorded within CONNECT_(—) REQUEST_TIMEOUTdays of a previous one.

[0101] Table 6, T_SERVICE_TYPE, identifies a particular type of servicesupported through the router-router connection. TABLE 6 T_SERVICE_TYPEColumn Type Meaning or value ID Number This service ID-this is the sameas HG_PROP_SERVICE_TYPE in a message. DESCRIPTION String User-readabledescription of the service

[0102] Table 7, T_DEVICE, represents a device hosting a queue manager.T_DEVICE contains information used by several different entities. Thecolumns listed here are the only ones used by the router and its storedprocedures. TABLE 7 T_DEVICE Column Type Meaning or value ID NumberPrimary key-the device ID CONNECT_STATE Number Enumeration of possibledevice connection states: STATUS_OFFLINE- device is not connected to arouter STATUS_CONNECTING- a router is attempting to connect to thedevice STATUS_OFFLINE_(—) CREQUEST-device or other entity has requestedthat the device be brought online. STATUS_ONLINE_(—) CREQUEST-the routerthinks that the device is online (stale TCP connection). The device hassent a connection request indicating it wants to re-establish aconnection. STATUS_ONLINE-the device is online and can send and receivemessages STATUS_(—) DISCONNECTING_(—) CREQUEST-the router is attemptingto shut the device's stale socket before re-establishing the connection.ADDRESS Number The device's last known IP address PORT Number The devicelistens for connections from the router on this port. LAST_CONNECT_TIMEDate Last time the router successfully connected to the device.LAST_CONNECT_ATTEMPT Date Last time the router tried to connect to thedevice. MAC_ADDRESS2 Number The MAC address of the network card in thedevice. CONNECT_OID Number This is a sequence # that is used tocorrelate an update of the connect parameters with any subsequentupdates or selects DEVICE_GUID RAW(16) The computed device GUID. TheGUID contains the network ID, security, router ID (of the connectingrouter) and device ID. NETWORK_ID Number The network used to connect tothe device ROUTER_ID Number The router currently in charge of connectingto the device.

[0103] Table 8, T_CONNECT_REQUEST_IN is used by the router to transmit aset of connect requests to the database viaPKG_HGS_CONNECT.SP_HGS_CONNECT_REQUEST. TABLE 8 T_CONNECT_REQUEST_INColumn Type Meaning or value TRANSACTION_GUID RAW(16) All rows specificfor a particular invocation of SP_HGS_(—) CONNECT_REQUEST are identifiedby the router by this GUID. MAC_ADDRESS Number The MAC address of thecon- necting device (can be NULL) IP_ADDRESS Number The IP address ofthe listener socket on the device PORT Number The port number of thelistener socket NETWORK_ID Number The network ID of the network used totransmit the connect request DEVICE_ID Number The device ID of thedevice making the connect request (can be NULL)

[0104] Table 9, T_HGS_CONNECT_ACTIVITY_IN, is used by the router toinform the database of the set of devices whose connect states havechanged. The router populates this table and callsPKG_HGS_CONNECTION.SP_HGS_CONNECT_ACTIVITY to process the inserted rows.TABLE 9 T_HGS_CONNECT_ACTIVITY_IN Column Type Meaning or valueTRANSACTION_GUID RAW(16) All rows specific for a particular invocationof SP_HGS_(—) CONNECT_ACTIVITY are identified by the router by thisGUID. DEVICE_ID Number The device ID of the connected or disconnecteddevice STATE Number Either STATUS_ONLINE or STATUS_OFFLINE-indicates thenew device state. ROUTER_ID Number ID of the router previously connectedto the device

[0105] Table 10, T_MESSAGE, contains the routing and deliveryinformation for a message. TABLE 10 T_MESSAGE Column Type Meaning orvalue ID Number Primary key GUID RAW(16) This is the message ID and isneeded for correlating the message with receipts. SEND_STATE NumberSTATUS_NOT_SENT- message has never been sent STATUS_SEND_IN_(—)PROGRESS-the router is attempting to send the message. STATUS_SEND_(—)FAILED-the router failed to send the message OID Number This is used tocorrelate changes in the send state with subsequent selects (in caseanother procedure updates the send state in the meantime)DESTINATION_DEVICE_ID Number The device ID of the device to receive themessage TIME_EXPIRED Date The time at which the message will expire-itshould not be delivered after this date and can be deleted. TIME_SENTDate The time at which the last send was attempted.

[0106] Table 11, T_PAYLOAD, is a database entry which contains a portionof a message. The router breaks a message into 256 byte chunks in orderto optimize use of space when uploading messages. A given message hasone T_MESSAGE row and usually 1-3, but sometimes up to 20 rows in theT_PAYLOAD table. TABLE 11 T_PAYLOAD Column Type Meaning or value ID (fkof T_MESSAGE.ID) Number Indicates the message associated with thispayload ITEM_INDEX Number Zero-based index of the payload chunk. This isused to order the chunks when reassembling them. DATA RAW(256) The datain the chunk

[0107] Table 12, T_MESSAGE_ACTIVITY_IN, is used by the router to informthe database of messages sent and not sent. The router creates atransaction GUID and puts it in each row of T_MESSAGE_ACTIVITY_IN, thencalls PKG_HGS_MESSAGE.SP_HGS_RECORD_MESSAGE_ACTIVITY to process theresults. TABLE 12 T_MESSAGE_ACTIVITY_IN Column Type Meaning or valueTRANSACTION_GUID RAW(16) All rows specific for a particular invocationof SP_HGS_RECORD_MESSAGE_ACTIVITY are identified by the router by thisGUID. MESSAGE_GUID RAW(16) The message ID (HG_PROP_MESSAGE_ID) for themessage WAS_SENT Number Non-zero if sent, zero if not sent

[0108] 3.2 Router Stored Procedures

[0109] The router's stored procedures are contained in three packages,

[0110] PKG_HGS_ROUTER—for configuring the router and bringing it online.

[0111] PKG_HGS_CONNECTION—for processing device connections to a router.

[0112] PKG_HGS_MESSAGE—for processing messages to a device.

[0113] 3.2.1 Router Online and Offline States

[0114] PKG_HGS_ROUTER contains the stored procedures that bring a routeronline, that take the router offline, that reset the watchdog timer, andthat find other routers. A generalized router state diagram for theseprocedures is illustrated in FIG. 5. The states of the router deviceinclude an offline state 1500, a running state 1510, and a disconnectscheduled state 1520. The PKG_HGS_ROUTER software package contains theseand other stored procedures. For example:

[0115] SP_HGS_INIT—The router calls this stored procedure when it comesonline. The procedure has five purposes:

[0116] To create a new entry in T_ROUTER for routers that are connectingfor the first time.

[0117] To initialize device states (if a network has only one router andthe router terminates without calling SP_HGS_EXIT, device connect statesmay hold the erroneous STATUS_ONLINE at the time SP_HGS_INIT is called.

[0118] To mark the router as online.

[0119] To create new entries in the T_ROUTER_NIC table for new networkcards.

[0120] To record the current IP addresses of the router's network cards.

[0121] To return the configuration information for the router's networkcards.

[0122] A router creates one row in the T_ROUTER_NIC_IN table for each ofits network cards. All rows should contain the same transaction GUID.This GUID is passed into SP_HGS_INIT to update its T_ROUTER_NIC table.

[0123] The following are the parameters for calls to SPS_HGS_INIT:Parameter List 1. PKG_HGS_ROUTER.SP_HGS_INIT Column Type Meaning orvalue TRANSACTION_ID_PARAM RAW All rows of T_ROUTER_NIC_IN (in) specificfor a particular invocation of SP_HGS_INIT are identified by the routerby this GUID. HOST_NAME_PARAM String The DNS host name for the callingrouter (in) MAC_ADDRESS_PARAM Number The MAC address of the network cardused by the (in) router's queue manager-this is used to find therouter's device ID in T_DEVICE. ROUTER_ID_PARAM Number The router's ID.(out) (T_ROUTER.ID) TIME_PARAM Date The database's notion of the currenttime. (out) CURS_ROUTER_NIC_PARAM Cursor This cursor contains one rowper network card: the cursor (out) contains the configurationinformation for that card.

[0124] CURS_ROUTER_NIC_PARAM schema Column Type Meaning or valueNIC_INDEX Number Zero-based index of the network card. NETWORK_ID (fk ofNumber The network card is on T_NETWORK.ID) this network. IP_ADDRESSNumber The router should use this IP address to bind to the networkcard. DEVICE_PORT Number The router should listen for connect requestson this port (may be NULL if no listener is configured on this card)ROUTER_PORT Number The router should listen for connections from otherrouters on this port (may be NULL if no listener is configured on thiscard. Should be NULL in general if card is not connected to the datacenter network). MC_AVAILABILITY_ADDRESS Number The multicast address tobe used to transmit the router availability multicast (can be NULL)MC_AVAILABILITY_PORT Number The multicast port to be used to transmitthe router availability multicast (can be NULL) MC_AVAILABITY_FREQUENCYNumber # of (fractional) days between router avail- ability multicasts(can be NULL)

[0125] SP_HGS_WATCHDOG—this stored procedure has several differentfunctions:

[0126] It records the fact that the router is still active.

[0127] It updates the router's load metric and adjusts network cardconfiguration based on this metric.

[0128] It takes routers that are inactive (e.g. because they terminatedunexpectedly, were isolated from the database, were deadlocked) offline.This also marks all devices assigned to the inactive router as offline.

[0129] It transmits the router's network card configuration, allowingthe router to update any changes.

[0130] The stored procedure has the following parameters: Parameter List2. PKG_HGS_ROUTER.SP_HGS_WATCHDOG Parameter Type Meaning or valueROUTER_ID_PARAM Number The ID of the calling (in) router (T_ROUTER.ID)LOAD_METRIC_PARAM Number This is a calculated (in) metric based on therouter's performance. Higher numbers indicate that a router is moreheavily loaded. GO_OFFLINE_PARAM Number The router should bring (out)itself offline if this parameter is non-zero upon return. TIME_PARAMDate The database's notion of (out) the current timeCURS_ROUTER_NIC_PARAM Cursor This is the same as that (out) inSP_HGS_INIT

[0131] Note in particular from the above that each router periodicallydetermines a relative load metric and stores this information in theLOAD_METRIC. In the preferred embodiment, a lower number indicates abetter performing router. As will be understood shortly, the LOAD_METRICentry is used by the router to determine how often to send anavailability message to the network devices 10.

[0132] SP_HGS_EXIT—this stored procedure is called as its last databasecommunication before terminating. It has the following functions:

[0133] It marks the router as offline.

[0134] It sets the device connect state of any devices connected to therouter as offline. SP_HGS_EXIT has the following parameters: ParameterList 3. PKG_HGS_ROUTER.SP_HGS_EXIT Parameter Type Meaning or valueROUTER_ID_PARAM Number The router going offline (T_ROUTER.ID)

[0135] SP_HGS_FND_PATH—this stored procedure finds the possible pathsbetween two routers. It has the following parameters: Parameter List 4.PKG_HGS_ROUTER.SP_HGS_FIND_PATH Parameter Type Meaning or valueSRC_ROUTER_ID_PARAM Number The router ID of the (in) calling router(T_ROUTER.ID) DEST_ROUTER_ID_PARAM Number The router to connect via (in)the router-router protocol (T_ROUTER.ID) CURS_PATH_PARAM Cursor Thiscursor contains rows (out) describing how to connect to the destinationrouter. Each row describes a possible path.

[0136] Parameter List 5. CURS_PATH_PARAM schema Colunm Type Meaning orvalue BIND_ADDRESS Number The IP address to be used to bind to a networkcard in the router IP_ADDRESS Number The IP address of a network card onthe destination router PORT Number The port # used by the destination tobind a router- router listener. The calling router should connect tothis port. NETWORK_ID Number The ID (T_NETWORK.ID) of the network usedto connect the two routers. SECURE Number The security level(T_NETWORK.SECURE) of the above network.

[0137] 3.2.2 Router Handling of Connection Requests

[0138] The routers also of course handle connection requests from thenetwork devices 10. A state diagram for this process is shown in FIG. 6.Generally, a state 1600 is an offline state, state 1610 is entered whena connection request is received in the offline state, state 1620 is anonline state, state 1630 is an online connection request state, andstate 1640 is entered when the router is disconnecting and receives aconnection request.

[0139] The procedures called to implement these states are nowdiscussed. PKG_HGS_CONNECTION contains the stored procedures to recordconnection requests from devices, to inform routers of devices requiringconnection and to record the connection state of these devices.PKG_HGS_CONNECTION has the following stored procedures that are calledfrom the C++ router:

[0140] SP_HGS_CONNECT_REQUEST—used by the router to transmit the set ofdevices issuing connect requests.

[0141] SP_HGS_CONNECTION_ACTIVITY—used by the router to transmit the setof devices whose connect states has changed.

[0142] SP_HGS_CONNECT—returns the set of devices requiring connectionbecause of connect requests.

[0143] SP_HGS_MSG_CONNECT—returns the set of devices requiringconnection because of messages pending.

[0144] SP_HGS_DISCONNECT—returns the set of devices requiringdisconnection from stale connections.

[0145] PKG_HGS_CONNECTION also contains a stored procedure,SP_HGS_ONE_CONNECT_REQUEST, that may be called externally by otherstored procedures to bring a device online. This may be done to preparethe device to receive messages from a service.

[0146] SP_HGS_CONNECT_REQUEST—This stored procedure records connectionrequests from devices. Devices send their IP address and listener portwhen they have messages to send to a router. The stored procedurerecords these in the database. The router creates rows in theT_CONNECT_REQUEST_IN table, then calls SP_HGS_CONNECT_REQUEST to processthe requests. SP_HGS_CONNECT_REQUEST has the following functions:

[0147] It creates a new row in the T_DEVICE table for unassigneddevices. SP_HGS_CONNECT_REQUEST will do this for devices which do nottransmit a MAC address or device ID (indicating that they don't knoweither quantity) or that transmit MAC addresses or device Ids thataren't in the T_DEVICE table.

[0148] It updates the IP address and port # in each device's T_DEVICErow.

[0149] It assigns a router and device GUID to a device based on routerload and the connecting network.

[0150] SP_HGS_CONNECT_REQUEST has the following parameters: ParameterList 6. PKG_HGS_CONNECTION.SP_HGS_CONNECT_REQUEST Parameter Type Meaningor value TRANSACTION_GUID RAW(16) This selects rows from theT_CONNECT_REQUEST_IN table. The stored procedure processes, then deletesthese rows.

[0151] SP_HGS_CONNECT. The router calls this stored procedure to get theset of devices requiring connection because of connection requests. Thisincludes connection requests from SP_HGS_CONNECT_REQUEST and those dueto messages being sent to offline devices. These are cases requiringrelatively immediate response. SP_HGS_CONNECT returns a cursorcontaining the information needed to connect. The procedure has thefollowing parameters: Parameter List 7.PKG_HGS_CONNECTION.SP_HGS_CONNECT Parameter Type Meaning or valueROUTER_ID_PARAM Number The router ID of the router requesting deviceconnection information. CURS_HGS_CONNECT_PARAM Cursor This cursorcontains one row per device needing connection.

[0152] Parameter List 8. CURS_HGS_CONNECT_PARAM schema Column TypeMeaning or value DEVICE_GUID RAW(16) The device GUID of the devicerequiring connection (T_DEVICE.DEVICE_GUID) NETWORK_ID Number The routershould connect to the device through a port bound to a card attached tothis network. IP_ADDRESS Number The IP address to connect to (thedevice's listener socket is bound to this address) PORT Number The port# of the device's listener.

[0153] SP_HGS_MSG_CONNECT—this stored procedure returns the set ofconnections to be made to devices with messages pending. This callshould be made less frequently than SP_HGS_CONNECT (and, if possible,with lower priority) because it is relatively expensive compared toSP_HGS_CONNECT and because the connections do not need to be made in atimely fashion. SP_HGS_MSG_CONNECT has the same parameter signature asSP_HGS_CONNECT.

[0154] SP_HGS_DISCONNECT—this stored procedure returns the set of staledevice connections. The router should attempt to disconnect from thesedevices. SP_HGS_DISCONNECT has the same parameter signature asSP_HGS_CONNECT.

[0155] SP_HGS_CONNECT_ACTIVITY—the router updates the connection statein T_DEVICE using this stored procedure. The router inserts rows intoT_HGS_CONNECT_ACTIVITY_IN. Each row contains a transaction GUID whichcorrelates the row with the particular invocation ofSP_HGS_CONNECT_ACTIVITY. SP_HGS_CONNECT_ACTIVITY updatesT_DEVICE.CONNECT_STATE for each row processed, deletes the row and sendsa result code in CURS_ACTIVITY_PARAM which is returned from the storedprocedure. Parameter List 9. PKG_HGS_CONNECTION.SP_HGS_CONNECT_(—)ACTIVITY Parameter Type Meaning or value TRANSACTION_(—) RAW All rows inID_IN (16) T_HGS_CONNECT_ACTIVITY_IN are identified by the router bythis GUID. CURS_ACTIVITY_(—) Cursor The procedure returns one row perPARAM row in T_HGS_CONNECT_ACTIVITY_IN to indicate success or failure ofthe operation.

[0156] Parameter List 10. CURS_ACTIVITY_PARAM schema Column Type Meaningor value DEVICE_ID Number The ID (T_DEVICE.ID) of the device whose statehas changed RESULT Number ERROR_NONE (=0) if the row was properlyformed, ERROR_NO_SUCH_DEVICE (=1) if the device ID did not match any inthe T_DEVICE table. ERROR_BAD_STATE if T_HGS_CONNECT_ACTIVITY_IN. STATEwas not STATUS_ONLINE or STATUS_OFFLINE

[0157] SP_HGS_ONE_CONNECT_REQUEST—this stored procedure makes a connectrequest on behalf of some other stored procedure. It operates similarlyto SP_HGS_CONNECT_REQUEST (in fact it provides the implementation forSP_HGS_CONNECT_REQUEST in the current, but not subsequent, code base).It has the following parameters: Parameter List 11.PKG_HGS_CONNECTION.SP_HGS_ONE_CONNECT_REQUEST Column Type Meaning orvalue DEVICE_ID_PARAM Number The device requiring connection(T_DEVICE.ID) (may be NULL) MAC_ADDRESS_PARAM Number The MAC address ofthe device requiring connection (may be NULL) ADDRESS_PARAM Number IPaddress of the device requiring connection (may be NULL if device ID orMAC address correctly specified) PORT_PARAM Number Listener port # (maybe NULL, see ADDRESS_PARAM) NETWORK_ID_PARAM Number Network ID to beused to communicate to above address (may be NULL, see ADDRESS_PARAM)

[0158] 3.3.3 Router Messaging States

[0159] Once connections are made, the routers 250 of course also handlethe processing of messages. This process is shown generally in FIG. 7,and includes a status not sent state 1700, a status sending state 1710,a status failed sending state 1720, and a status message deleted state1730.

[0160] The stored procedure PKG_HGS_MESSAGE contains the program codethat verify incoming messages, that pick messages eligible fortransmission, that update message state, and that delete messages. Thefollowing messages are intended for external access:

[0161] SP_HGS_PUT_MESSAGES—this procedure verifies the destinationaddress of messages and commits the insert transaction.

[0162] SP_HGS_GET_MESSAGES—this procedure gets a cursor of payloads ofmessages to be sent from a particular router to connected devices.

[0163] SP HGS RECORD MESSAGE ACTIVITY—this procedure reports the resultsof attempts to send the messages retrieved by SP_HGS_GET_MESSAGES

[0164] SP_HGS_DELETE_EXPIRED—this procedure deletes messages that haveexpired. It should be run from a job within Oracle.

[0165] Both SP_HGS_GET_MESSAGES and SP_HGS_RECORD_MESSAGE_ACTIVITYupdate T_MESSAGE.SEND_STATE. Each of them updates T_MESSAGE.OID wheneverit updates T_MESSAGE.SEND_STATE. This allows each stored procedure toexclude rows modified between selection via cursor and update.

[0166] SP_HGS_PUT_MESSAGES—the router creates a row in the T_MESSAGE andmultiple rows in the T_PAYLOAD table for each message persisted to thedatabase. It uses a unique OID to mark all of these messages and uniqueIDs to mark each individual message and its payloads.SP_HGS_PUT_MESSAGES validates the destination device IDs—these arecreated by C++ applications and may be invalid. It deletes any invalidmessages and commits the rest. Parameter List 12.PKG_HGS_MESSAGE.SP_HGS_PUT_(—) MESSAGES Parameter Type Meaning or valueOID_PARAM Number All messages to be processed are marked with this OID

[0167] SP_HGS_GET_MESSAGES—the router retrieves the set of messages toprocess via this stored procedure. The stored procedure returns a cursorof payloads; these payloads are ordered by message ID and then bypayload item index. SP_HGS_GET_MESSAGES changes the message state toSTATUS_SEND_IN_PROGRESS for outgoing messages to prevent a resend.SP_HGS_GET_MESSAGES has the following parameters: Parameter List 13.PKG_HGS_CONNECTION.SP_HGS_GET_(—) MESSAGES Parameter Type Meaning orvalue ROUTER_ID_PARAM Number Calling router's ID CURS_PAYLOAD_REF_(—)Cursor The payloads of the messages to PARAM be sent.

[0168] Parameter List 14. CURS_PAYLOAD_REF_PARAM schema Column TypeMeaning or value MESSAGE_GUID RAW(16) HG_PROP_MESSAGE_ID from themessage-this is used to correlate messages, acks and receiptsDEVICE_GUID RAW(16) The GUID of the destination device ITEM_INDEX NumberThe zero based index of the payload chunk. Each chunk but the last is256 bytes long. They are combined to form a whole message. DATA RAW(256)The payload data.

[0169] SP_HGS_RECORD_MESSAGE_ACTIVITY—this stored procedure records theresult of an attempt to send a message retrieved viaSP_HGS_GET_MESSAGES. The router inserts rows into T_MESSAGE_ACTIVITY_INindicating the results of a transfer attempt. It marks each row in thistable with a transaction GUID which it passes intoSP_HGS_RECORD_MESSAGE_ACTIVITY.SP_HGS_RECORD_MESSAGE_ACTIVITY deletesany messages marked as sent and sets the send state of any unsentmessages to STATUS_SEND_FAILED.SP_HGS_RECORD_MESSAGE_ACTIVITY has thefollowing parameters: Parameter List 15.PKG_HGS_MESSAGE.SP_HGS_RECORD_MESSAGE_ACTIVITY Column Type Meaning orvalue TRANSACTION_GUID RAW(16) All rows in T_MESSAGE_ACTIVITY_INspecific for a particular invocation are identified by the router bythis GUID.

[0170] 4.0 Device Connection Protocol

[0171] Having now some basic appreciation for the various informationmaintained to effect message routing, the following mechanisms are usedto allow network devices to send connection request messages in anattempt to communicate with the data center through the routers 250 inaccordance with the invention.

[0172] Basically, there are three possible scenarios for a networkdevice attempting to connect to a router, including broadcast requests,DNS (static IP) requests, and mulficast type requests.

[0173] Broadcast: A device may broadcast its device connection packet incertain limited instances, such as if it is less than one network hopfrom a router.

[0174] DNS or static IP: A device may send a connection packet to arouter known to be at a particular DNS or static IP address.

[0175] Multicast or broadcast availability: This is the most common caseand the one to which the present invention is directed. A routerannouncer process multicasts or broadcasts a list of routers that can besent connection requests. The multicast or broadcast takes place on aknown IP address and port, using a UDP protocol. The payload portion ofsuch a router announcement service UDP packet is shown in FIG. 8. Thepacket includes at least an identifier field 500 indicating the type ofpacket, e.g., that this is a router announcement packet. A field of 128bits is allocated for the identifier field 500 in this embodiment.

[0176] In addition, a time field 510 indicating the time of theannouncement, and a port number 520 for establishing a connection to therouter, are also included. The time field 510 supports synchronizationof events within the entire system as well as security functions. Theport number provides the port used to address the packet to the routerprocess. A separate network address for the router need not be specifiedin the payload portion of the packet, since this information can begleaned from the UDP header information (not shown in FIG. 7).

[0177] The system allows provisioning for more than one router asequally preferred. For instance, if two routers are at a particularlocation, then they can each send availability messages. Devices wouldbe as likely to receive one packet as the other. The preferred portnumber of the router announcer is 18505. The preferred port forconnection requests on the router is 18503.

[0178] In response to receiving an announcer message, the networkdevices can then request that they be permitted to connect to theannouncing router. This takes the form of a device connection UDPpacket. The packet itself contains enough information to discern atleast the requesting device's IP address.

[0179] There are three cases for the device connect messages shownrespectively in FIGS. 9A, 9B, and 9C.

[0180] In the first instance, shown in FIG. 9A, the network device doesnot know its device ID or MAC address. This can be used as an initialprovisioning case for devices with inaccessible MAC addresses.

[0181] In a second instance, shown in FIG. 9B, the network device knowsits MAC address. This is the preferred case as it allows the server tochange the device's device ID.

[0182] In a third instance, shown in FIG. 9C, the device has cached itsdevice ID from a previous call.

[0183] Regardless of the addressing format, the payload portion of thepacket data provides the port number and device ID or MAC address forthe device, as used by the router in establishing the connection to therequesting network device.

[0184] Finally, in response to receipt of one of these messages, a givenone of the routers will respond by connecting to the network device 10.The router preferably sends a clock message as the first message to thedevice. The clock message contains the network device's assigned GUID inthe message header field,. The device can then use this GUID as itssender identification for subsequent messages. The clock message cancontain a device ID different from the one sent in the case where theMAC address is unknown. The device will persist the new device ID anduse it in subsequent device connections.

[0185] The particular router 250 chosen for response can be coordinatedby the queue manager 260 or in other ways, by taking into account theloading factors of the respective routers 250. For example, a relativelylightly loaded router will be selected for handling the new connection,as opposed to a presently busier one. Round robin, least loaded, or anynumber of other known load balancing schemes can be employed to selectamong the available routers 250.

[0186] While this invention has been particularly shown and describedwith references to preferred embodiments thereof, it will be understoodby those skilled in the art that various changes in form and details maybe made therein without departing from the scope of the inventionencompassed by the appended claims.

What is claimed is:
 1. In a network system that connects a centrallocation with a large number of network devices, a method for balancingthe coordination of responses to connection requests originating at thenetwork devices comprising the steps of: sending a router availabilitymessage through a designated router to a specific network device, thedesignated router located at the central location; sending a connectionrequest message from a specific network device to the central locationthrough the router indicated by a received router availability message,and only in response to receiving a router availability message at thespecific network device; forwarding the connection request message tothe central location; and reassigning connection requests from aspecific network device to a router under control of the centrallocation, so that connection requests remain distributed among a numberof available routers.
 2. A method as in claim 1 wherein the routeravailability message is repeated at a rate which depends upon therelative availability of the router to service the message.
 3. A methodas in claim 1 wherein the messages are connection request messagesoriginating from a network devices requesting that a connection be madeto the central location.
 4. A method as in claim 1 wherein.the routeravailability message indicates a network address at which the networkdevice may respond.
 5. A method as in claim 1 wherein the routeravailability message is sent as a multicast message to a specific groupof network devices.
 6. A method as in claim 1 wherein the routeravailability message represents a right for the network device toconnect to the router for a limited time only.